fix(benchmarks): remove hidden Farkas evidence cap - #1333
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7cbf8cd142
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| buffer += decoder.decode(block) | ||
| if buffer[:1] in _JSON_WHITESPACE: | ||
| buffer = buffer.lstrip(_JSON_WHITESPACE_CHARS) | ||
| try: | ||
| value, end = parser.raw_decode(buffer) |
There was a problem hiding this comment.
When a schema-valid evidence object contains large legal whitespace between tokens (rather than only at the beginning or end covered by the new test), this loop retains the entire prefix and reruns raw_decode over it after every 64 KiB read. Runtime therefore grows quadratically: a 64 MiB string/prefix already takes about 30 seconds locally, so a roughly 128 MiB artifact within the task's 4 GiB storage allowance can exceed the 120-second verifier timeout and produce no deterministic reward. Parse the bounded evidence schema incrementally, including discarding whitespace at token boundaries, so removing the unpublished byte cap does not replace it with a practical timeout cap.
AGENTS.md reference: AGENTS.md:L180-L183
Useful? React with 👍 / 👎.
2 participants
Summary
This is one bounded benchmark-integrity change based on an audit of historical PR #876 against the current Harbor architecture at
413e3846c2529af9e8f90a96d6f5679b23ed04f2.The audited Farkas task accepted schema-valid evidence with arbitrary JSON whitespace in its public contract, but the hidden verifier rejected files above 1 MiB. This removes that unpublished cap and ports the task-local loader to the repository's existing streaming JSON pattern. No product/runtime behavior, evaluator strategy, or general agent skill is added.
Integrity audit
exact-farkas-ldl-slicehad a hidden 1 MiB cap. This PR fixes that one task.task.toml; timeout/error states remain non-conclusions.Frozen task sample
Source SHA:
413e3846c2529af9e8f90a96d6f5679b23ed04f2exact-farkas-ldl-slice:sha256:5ff1eed2a5ff443222d181f3c676ac5012b4cd01189f1d983ef2e5256e447848polynomial-map-collision:sha256:9243ca8ee4800cb8eae039a08e0414c7fcfd8e111c3d1a3c8d6fbbfc44884abdrp2-homology-lattice:sha256:35c60550453aa1979b4694a399813023844ab3a6524643e106a9e983836b6539apollonius-gap-repair:sha256:701b5fc809ec6973a55d8a30568d89829c9c7f683b80c04241a4df2d6bc61636elementwise-fixed-no-global-invariant:sha256:ce3612dd2ebff71eb14234c965ef1d8959e19e43f0953b3366df13f04446f2aeProspective changed-task digest:
sha256:174ddb8eb0f35a81f21a196aaa7dc562f5af399278da763b5bc6f7f5c0bf2e58.Pilot evidence
The planned paired pilot was gpt-5.6-luna at medium reasoning, with a no-Jacobian control and a current-Jacobian-MCP treatment, and no general evaluation skill exposed. Both substantive attempts ran in named tmux sessions.
The host has no Docker daemon, so a Bubblewrap clean room was used after proving that the repository and verifier files were not visible. The attempt was stopped after repeated infrastructure failures: the installed standalone Codex package lacked its code-mode host executable inside the clean room, so the model could not read or write the task workspace. The control produced only a blocker trajectory (59,307 input, 52,992 cached input, 552 output, 142 reasoning tokens); neither condition produced a mathematical submission. The comparison is therefore INCONCLUSIVE, not a treatment result. It provides no evidence for adding a general tool-use skill, and this PR adds none.
Change
MAX_EVIDENCE_BYTESrejection.Validation
make harbor-plan BASE=origin/main: selects only the Farkas host leaf, its generic verifier-contract leaf, and its Oracle.git diff --check origin/main...HEAD: passed.make check: Ruff, formatting, complexity, mypy, and 869/870 unit tests passed. The remaining unrelatedtest_python_distribution_identity_binds_installed_file_bytesfailed due same-timestamp metadata caching; its isolated reproduction failed once and passed immediately on retry without a code change, and the full retry hit the same pre-existing timing condition./bin/sh: docker: not found), so no Oracle verdict is claimed.Deferred proof gaps
A Docker-capable host should run the selected exact-task Oracle and the frozen five-task paired pilot. Those are explicit evidence gaps; they do not broaden this PR beyond the single reproducible verifier-contract defect.